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Abstract 


Besides measures of students’ performances, a valid assessment of students’ efficiency in a 
teaching process should also include measures of invested mental effort. The research presented 
herein covers several approaches in measuring students’ mental effort including 5, 7 and 9-point 
Likert type scales, time on task, as well as eye tracking technique which in combination with 
performance measures provide valid information on students’ success. Results of the research 
showed good correlation between mental effort assessed by 7 and 5-point Likert type scales and 
students’ performance, while the use of the 9-point scale showed a low degree of correlation, 
thus recommending the use of a scales with 5 and 7 points for educational purposes over 9-point 
scales. The research presented herein illustrates how eye tracking can be used to support the 
evaluation of invested mental effort. Additionally, this method enabled the identification of some 
student difficulties in the analyzed area — Stereochemistry. 
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Introduction 


Over the years a vast majority of researchers performed their research on students’ 

efficiency in teaching process relying only on measures of students’ performances. 
However, although such measures provide valuable information about the effectiveness 
of the teaching process, they cannot be its only indicator. It is found that one also has 
to take into account the basic idea of Cognitive Load Theory (Paas & van Merrienboer, 
1993) about the limited capacity of the working memory. Accordingly, research showed 
an increasing interest in cognitive load measures in recent years, and in particular, 
measures of its important construct — mental effort. 
There have been developed various approaches to measuring mental effort. The main 
classification of mental effort measures implies subjective and objective approach. The 
objective approach is mainly based on physiological and behavioural measures. Some 
of the most common techniques used within this framework are eye tracking technique 
(Baluyut & Holme, 2019; Tang & Pienta, 2012), brain activity measurement (Whelan, 
2007) and measurements of cardiovascular indicators (Paas & Merrienboer, 1994). 

Subjective measures are the most frequently applied empirical methods for 
measuring mental effort in science education research. They are based on the assumption 
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that people are able to revise their cognitive processes and assign a numerical value to 
the invested mental effort (Paas et al., 2003). Subjective measures are usually based on 
the use of scales such as the Likert type scale. Their usage as a reliable measurement 
instrument for assessing the invested mental effort is very-well documented in the 
literature (Ayres, 2006; Kalyuga, Chandler, & Sweller, 2000; Kalyuga, Chandler, & 
Sweller, 2001; Tindall-Ford, Chandler, & Sweller, 1997). The above-mentioned authors 
suggest that such measures are the most reliable and most sensitive to detect relatively 
small differences in mental effort. In addition, such measures highly correlate with 
objective measures. 

Modern trends in the assessment of efficiency include both performance and 

mental effort measures. One of the affirmed approaches in modelling the relation between 
performance and mental effort was proposed by Paas and van Merrienboer (1993). This 
model made it possible to determine, in a relatively simple way, the relation between 
performance and mental effort, that is, the combined effect of these two indicators on 
students’ efficiency. 
The aim of this research was to compare various approaches in measuring students’ 
mental effort (subjective — Likert type scales and objective: time on task and eye- 
tracking) as well as to combine them with measures of performance to provide valid 
information on students’ efficiency. 


Research Methodology 


The research was carried out in several stages: 


(1) Application of 5, 7- and 9-point Likert type scales and determining which 
of the scales is the most sensitive in evaluating the mental effort. 

(1i) Measuring the time necessary for the task to be solved. 

(iii) Employing the eye tracking methodology to provide information on the 
invested mental effort. 

(iv) Combining measures of performance and mental effort, thus providing 


information about instructional efficiency (main study). 

Different stages of the research included different research samples. Thus, the 
first stage of the research included 62 secondary school students aged 15 to 16. The 
second and third included 17 students majoring in chemistry teaching from the Faculty 
of Sciences, University of Novi Sad, who were in their final year of the Bachelor studies. 
The last (main) stage included 189 secondary school students from Novi Sad (Serbia). 

Research instruments were designed according to the objectives of individual 
studies. The first stage of research included 3 tests with 6 items, belonging to the teaching 
topic Dispersions. All three tests had analogous tasks, the design of which was identical, 
and the only difference could be found in numerical values and types of substances used. 
It is important to note that the first test included a 5-point Likert type scale, the second 
included 7-point scale, while the third included 9-point Likert type scale. This kind of 
design enabled a direct comparison of the sensitivity of the applied scales. The second 
and third stage of the research included an online test with 6 items, which belonged to 
the teaching topic Stereochemistry. The instrument used in the main stage included 15 
two-tier items, covering the topics: Group 14, 15, 16 and 17 Elements. 

All the participants, included in the research, accepted to willingly participate without 
any constraint or expectation of reward. All necessary permissions were obtained prior 
to research. 
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Research Results 


Results of the first stage of research showed that 7-point Likert type scale 
correlated best with students’ performance, while the 9-point scale showed poor 
correlation. Such results indicated justification of the use of 7-point scales in research 
designs. Poor correlation in the case of a 9-point scale could be explained by the fact 
that students were not able to evaluate fine differences in the mental effort (for instance 
nuances between very difficult and extremely difficult). A 5-point scale gave satisfactory 
correlation values which can be explained by the fact that students are familiar with such 
a way of evaluation due to the same way of grading (1 to 5) in the school system. These 
results are significant because they show that by selecting a scale of the appropriate range 
we allow students to reflect on their mental processes and to determine the amount of 
invested mental effort by assigning it a numerical value. In addition, these results support 
the application of subjective methods of evaluation of mental effort as convenient non- 
intrusive tools in educational research designs. 

The following stage of the research was focused on measuring the time needed to 
solve the task as well as on the results of eye tracking analysis. This powerful method 
enabled us to monitor the individual steps of the problem-solving procedure, to examine 
how much time is spent on various aspects of the task, as well as to keep track of revisited 
student’s fixations. This approach allowed us to determine what are the difficulties that 
students encounter within Stereochemistry, which impose the investment of a large 
amount of mental effort. Some of the identified difficulties involved the determination 
of the chirality of the molecules, as well as the absolute configuration of atoms in both 
acyclic and cyclic structures. 

Last but not least, combined measures of performance and mental effort provided 
valid information on instructional efficiency of several instructional designs which 
enabled inferring that the efficiency of instruction or individual in the teaching process 
cannot be considered separately from the Cognitive Load Theory. One of the instructions 
that proved to be very effective in mastering chemical concepts is the instruction based 
on the Triplet model. Namely, using combined measures of students’ success, it was 
found that this instruction increased student performance and reduced amount of invested 
mental effort. 


Conclusions and Implications 


The results of the research confirmed the expectations of the theoretical 
background that is, the need to include parameters related to the Cognitive Load Theory 
in efficiency evaluation. Instructional efficiency represents an important area of science 
education research and therefore, research based on the improvement of the methods and 
techniques of its evaluation is highly requisite. 

In addition to the described results, further research should be focused on the 
application of the eye tracking technique as a very powerful tool which, in addition to the 
described benefits, also provides very important information on students’ misconceptions. 
Besides, tested parameters should also include information on students’ motivation as an 
important indicator of students' success. 


| 


Proceedings of the 3“ International Baltic Symposium on Science and Technology Education, BalticSTE2019 | 


180 


Acknowledgements 


Presented results are part of the research conducted within the Project “The 
quality of education system in Serbia from European perspective”. Grant No. 179010 
of the Ministry of Education, Science and Technological Development of the Republic 
of Serbia. 


References 


Baluyut, J. Y., & Holme, T. A. (2019) Eye tracking student strategies for solving stoichiometry 
problems involving particulate nature of matter diagrams. Chemistry Teacher International. 
Advance online publication. doi:10.1515/cti-2018-0003. 

Tang, H., & Pienta, N. (2012). Eye-tracking study of complexity in gas law problems. Journal of 
Chemical Education, 89, 988-994. doi:10.1021/ed200644k. 

Whelan, R.R. (2007). Neuroimaging of cognitive load in instructional multimedia. Educational 
Research Review, 2, 1-12. doi:10.1016/j.edurev.2006.11.001. 

Paas, F. G. W. C, & van Merriénboer, J. J. G. (1994). Variability of worked examples and transfer 
of geometrical problem-solving skills: a cognitive-load approach. Journal of Educational 
Psychology, 86, 122-133. doi:10.1037/0022-0663.86.1.122. 

Paas, F., Tuovinen, J. E., Tabbers, H., & van Gerven, P. W. M. (2003). Cognitive load measurement 
as a means to advance cognitive load theory. Educational Psychologist, 38, 63-71. 
doi:10.1207/S15326985EP3801_ 8. 

Ayres, P. (2006). Using subjective measures to detect variations of intrinsic cognitive load within 
problems. Learning and Instruction, 16, 389-400. doi:10.1016/j.learninstruc.2006.09.001. 

Kalyuga, S., Chandler, P., & Sweller, J. (2000). Incorporating learner experience into the 
design of multimedia instruction. Journal of Educational Psychology, 92, 126-136. 
doi:10.1037/0022-0663.92.1.126. 

Kalyuga, S., Chandler, P., & Sweller, J. (2001). Learner experience and efficiency of instructional 
guidance. Educational Psychology, 21, 5—23. doi:/10.1080/01443410124681. 

Tindall-Ford, S., Chandler, P., & Sweller, J. (1997). When two sensory modes are better than 
one. Journal of Experimental Psychology: Applied, 3, 257-287. doi:10.1037/1076- 
898X.3.4.257. 

Paas, F., & van Merriénboer, J. J. G. (1993). The efficiency of instructional conditions: An approach 
to combine mental effort and performance measures. Human Factors, 35, 737-743. doi:1 
0.1177/001872089303500412. 


